Quality and content evaluation of websites with information about immune checkpoint inhibitors: An environmental scan

Background Trustworthy educational information for patients is critical for increasing their knowledge base and preparing them for shared decision making with clinicians. As the internet has become an important source of health information for many patients, the purpose of this study was to assess the quality and content of websites with educational content about immune checkpoint inhibitors. Methods We performed an environmental scan of the currently available websites providing educational information for patients about immune checkpoint inhibitors. We used three search engines: Google, Bing, and Yahoo! (9/20/2021). Two independent investigators selected relevant uniform resource locators (URLs), appraised the quality of the websites, and collected their characteristics. We evaluated the accuracy, completeness, technical elements, design and aesthetics, readability, usability, and accessibility of the websites. The user experience was also evaluated. Results We identified 37 websites for analysis. In 10 websites (27%), it was not possible to know the source of the information provided. Thirty-three (89%) provided a definition with a simple explanation of cancer and treatment and 30 (81%) on complications of immune checkpoint inhibitors; only seven (19%) provided information about the balance between risks and benefits. Thirty-five (95%) provided a statement of purpose. Regarding the design, all 37 (100%) had appropriate visual aspects, typography, and grammar. Thirty-six (97%) were well organized. For most of the websites (n = 35, 95%) the content was easy to find. Only two websites had a readability score of 6, while the others had higher scores. Regarding the user experience, the overall quality of websites was rated as excellent in 16 (43%), good in 14 (38%), and fair in 7 (19%). Conclusions Our findings reveal that websites with information about immune checkpoint inhibitors mostly have general information about cancer, the treatments, and adverse events. Few websites provide information about the balance between harms and benefits of treatment, costs, the source of the information, or the hierarchy of evidence. These findings identify the gap in the quality and content of websites for patients treated with immune checkpoint inhibitors and can help website creators and developers.


Methods
We performed an environmental scan of the currently available websites providing educational information for patients about immune checkpoint inhibitors. We used three search engines: Google, Bing, and Yahoo! (9/20/2021). Two independent investigators selected relevant uniform resource locators (URLs), appraised the quality of the websites, and collected their characteristics. We evaluated the accuracy, completeness, technical elements, design and aesthetics, readability, usability, and accessibility of the websites. The user experience was also evaluated.

Results
We identified 37 websites for analysis. In 10 websites (27%), it was not possible to know the source of the information provided. Thirty-three (89%) provided a definition with a simple explanation of cancer and treatment and 30 (81%) on complications of immune checkpoint inhibitors; only seven (19%) provided information about the balance between risks and benefits. Thirty-five (95%) provided a statement of purpose. Regarding the design, all 37 (100%) had appropriate visual aspects, typography, and grammar. Thirty-six (97%) were well organized. For most of the websites (n = 35, 95%) the content was easy to find. Only two websites had a readability score of 6, while the others had higher scores. Regarding

Introduction
Immune checkpoint inhibitors (ICIs) are an increasingly used type of cancer immunotherapy that has been shown to improve the prognosis of patients with different types of cancers [1][2][3][4][5].
By blocking the function of key checkpoint factors, ICIs increase the activation of T cells, leading to a more vigorous response against tumors [6]. Monoclonal antibodies approved by the U.S. Food and Drug Administration (FDA) for cancer treatment include the programmed cell death 1 (PD-1) inhibitors nivolumab, pembrolizumab, cemiplimab, and dostarlimab and the programmed death ligand 1 (PD-L1) inhibitors atezolizumab, durvalumab, and avelumab [6]. Another "brake" on immune cells that negatively regulates activation of T-cells (CD4+ and CD8+) is cytotoxic T-lymphocyte-associated protein 4 (CTLA-4); the antibody ipilimumab inhibits CTLA-4 function and is also used to treat cancer. Recently, the FDA has approved relatlimab, a checkpoint inhibitor that targets lymphocyte-activation gene 3 (LAG-3), for the treatment of melanoma. Immune checkpoint inhibitors have transformed dramatically cancer management. Moreover, more patients have become eligible for ICI therapy over the years, including about 43.6% of patients with cancer in 2018 [7]. However, not all cases respond equally to ICI treatment, and different predictors of response have been suggested (e.g., PD-L1 expression, tumor-infiltrating lymphocytes, tumor mutational burden, tumor-associated macrophages) [8,9]. Although ICIs have important benefits, they also can induce adverse events, called immune-related adverse events (irAEs), which affect different tissues of the body [10]. The irAEs are related to the activation of the immune system by ICIs. These adverse events might deteriorate patients' quality of life and may not be tolerable to some patients. When patients need to make a decision regarding the initiation of ICI treatment, they must balance these benefits with the harms associated with treatment, taking into consideration risk factors for developing these harms. Furthermore, in this decision process, it is important that patients consider their own values and preferences, including financial concerns.
The internet is a common source of health information for patients [11]. With the increased use of ICIs in cancer patients, it is important to understand the quality and content of educational websites providing information on this therapy for patients. The purpose of this study was to assess the quality and usability of available websites providing information about ICI for cancer patients.

Materials and methods
We performed an environmental scan to systematically find and appraise the quality of currently available websites that provide patients with information regarding ICIs.

Eligibility criteria
Considered for inclusion were websites created by US-based organizations, agencies, or companies that provided only English-language information for patients. We excluded websites providing only general information on cancer therapy or immunotherapy (not specific to immune checkpoint inhibitors), clinical practice guidelines from professional organizations, and information from medical journals, blogs, news items, and social media fora.

Information sources
An expert librarian performed a comprehensive search in three different engines: Google, Yahoo!, and Bing from inception to 9/20/2021. For the Google search, we used the results of the top 100 from "search with omitted results", and then we combined them with the results of "the most relevant results". Then, we excluded the duplicates.

Search strategy
The search terms used were: immunotherapy; "checkpoint inhibitors" OR "checkpoint inhibitor" (for Bing and Yahoo! we used the term checkpoint inhibitor); ipilimumab OR Yervoy OR pembrolizumab OR Keytruda OR nivolumab OR Opdivo OR atezolizumab OR Tecentriq OR durvalumab OR Imfinzi OR avelumab OR Bavencio OR cemiplimab OR Libtayo. To correctly perform the search, we signed off from personal accounts, cleared all history, and disabled safe search. For exporting the search engine results, we used SEOquake (www.seoquake.com), which is a free browser plugin that provides key search engine optimization metrics.

Selection process
Two reviewers (J.I.R. and G.S.) independently screened the retrieved URLs in DistillerSR (Evidence Partners) to determine eligibility of the websites. Disagreements were resolved by consensus or with the participation of a third party (M.L.O.) if necessary.

Quality assessment
The quality of each selected website was appraised independently by two reviewers (an internal medicine physician and a general medicine practitioner, J.I.R. and G.S., respectively). Discrepancies between the two reviewers were resolved with third party adjudication (M.L.O.).
We used previously validated standards for the development of health information resources on the internet [12,13]. Before assessing the websites, the reviewers spent time reviewing background information about the different types of ICIs, their mechanisms of action, and the cancer(s) targeted and familiarizing themselves with the evaluation tool. The tool covers seven domains: a) accuracy of the information (i.e., information based on guidelines, textbooks, opinion, or unknown), b) completeness (i.e., discussion of irAEs [i.e., type, management, and risk factors], considerations for special populations [history of autoimmune disease, patients with organ transplant, etc.], and considerations with vaccines), c) technical elements (i.e., disclosure of funding, privacy and data protection, and ownership of website), d) design and aesthetics (i.e., visual aspects of the website, quality of visual presentation, menu [i.e., directional icons, bars, indicators, listing, indexes], typography [e.g., type size at least 12 points, use of bold type, color, different sizes], appropriate grammar, consistent layout [i.e., illustrations adjacent to the related text, paragraphs with five or fewer sentences, line lengths of 30 to 50 characters and spaces, pages that do not appear cluttered], subheadings, relevant graphics and images, appropriate type of material [i.e. text, tables, equations, audio and video], and browser compatibility), e) readability to assess if each website was written at a 6th grade level or below (calculated using the SMOG readability formula [14]), f) usability (i.e., easy navigation [e.g., well organized, index, table of contents, site map, easiness to return to previous page, frequently asked questions], internal search engine, option to download/print materials, speed, content tagged, functionality [i.e., support content]), and g) accessibility (e.g., appropriate color contrasting, easy-to-find content).

User experience
Another reviewer (E.M.), who is a college student, assessed the user experience. We used a modified version of the US government's recommended usability questionnaire [15]. The modified questionnaire contains 10 items with five response options (from strongly agree to strongly disagree) assessing: 1) the ease of finding the required information, 2) number of steps to reach the desired information, 3) ease of returning back after a wrong link is clicked, 4) organization of the information on the site, 5) clear labeling of the links, 6) correctness of the destination reached after clicking a link, 7) ease of learning to use the website, 8) user satisfaction with the website, 9) fulfillment of the user's expectations, and 10) likelihood of recommending the website to others and returning back to the website in the future. We also incorporated questions previously used by Siddhanamatha et al. [16] to evaluate the perceived amount of information provided by the website about ICIs (too little, adequate, or too much), use of easily understandable words, overall rating of the website, and informativeness of the website for patients beyond information provided by health professionals.

Synthesis methods
Each of the items in the seven domains in the quality assessment was rated with one of three responses: yes, no, or partial. Each item received an absolute score of "1" if present or "0" if absent. For each item, we used frequencies and percentages to summarize items that met quality standards. The reading levels of the included websites were assessed using means and standard deviations. For each item of the user experience we calculated the number of websites for which the user responded "strongly agree" or "somewhat agree".

Results
Out of 1,083 URLs resulting from our search, we included 37 websites for analysis. After removal of duplicates, of the 874 URLs screened in the first step, most of them (n = 491, 56.2%) were excluded because the reviewers determined that the information provided by these websites was not directed specifically for patients. Sixty-eight (7.8%) websites were excluded because they had information about cancer treatment in general (n = 25) or about immunotherapy in general (n = 43), but not specific information about ICIs. Nine (1%) websites were excluded because they were unrelated to the topic. Fifty-two (5.9%) websites were excluded because they were duplicates. One hundred fifty-six (17.8%) websites were excluded because of format (they were YouTube videos, blog posts, social media posts, or newsletters, etc.). Nine (1%) websites were excluded because they were not created in the United States (S1 Table). The number of URLs identified in each step, the reasons for exclusions, and the final websites included are depicted in the flow diagram shown in Fig 1.

Website characteristics
The included websites are listed in Table 1. The distribution of domain extensions was as follows: 19 were .com (57.4%), 16 were .org (43.2%), and 2 were .gov (5.4%). For the pages that provided information in PDF format, we evaluated the main website (S2 Table).

Quality assessment
For each item assessed, the number of websites rated as "yes" is shown in Table 2. Below, we present a summary of the quality evaluation for each domain. S3 Table ranks the websites by overall score. Accuracy. The information provided by 24 websites (64.9%) was based on guidelines or other scientific publications. In 10 (27%) websites, it was not possible to know the source of the information. Of the 37 websites analyzed, 7 were from pharmaceutical companies manufacturing the immune checkpoint inhibitors (opdivo.com, yervoy.com, bavencio.com, keytruda.com, imfinzi.com, libtayo.com, tecentriq.com). The content provided by the pharmaceutical companies' websites and four other websites (cancerreaserch.org, gene.com, medicinenet.com, rxlist.com) was based on FDA information.
Completeness and comprehensiveness. Most of the websites (89.2%) provided a clear definition with a simple explanation of the topic ICI, and 56.8% provided information describing each treatment, how it works, mechanism of action, and contraindications. Most of the websites (81.1%) also provided information about complications. However, less than 20% of the websites had information about the balance between benefits and harms. The cost of the treatment was included in only 8.1% of the websites.
We added specific items to evaluate the completeness and comprehensiveness of the websites' information about ICIs. We judged that 74.7% of the websites provided information about irAEs. Specifically, most of the websites listed the irAEs (cardiac, dermatological, endocrine, gastrointestinal, hematological, muscular, neurological, ocular, pulmonary, renal, systemic and rheumatic). Less than half (40.5%) of the websites explained risk factors for irAEs. Most of the websites that provided information about the risk factors denoted pre-existing autoimmune disease and patients with organ transplant as the main risk factors. Few websites (n = 5, 13.5%) included information about the management of irAEs of ICIs. Only three websites provided information about vaccine contraindications their adverse events and benefits, or efficacy in patients receiving ICIs.
Technical elements. Most of the websites (94.6%) provided a clear statement of purpose, declaring that the information was not meant to replace the advice of a health professional. Moreover, most of the websites gave a clear definition of the target audience (91.9%). Thirty websites (81.1%) provided the date that the website was last updated. However, only three websites (8.1%) explained clearly the hierarchy of evidence, meaning that it was not clear what information was obtained from opinions and what information was from higher levels of evidence (e.g., cohort studies, randomized controlled trials, systematic reviews). The majority of the websites (78.4%) complied with advertisement guidelines [17].
Design and aesthetics. All the websites had: 1) appropriate visual aspects (scroll bars and alignment), 2) acceptable typography, and 3) appropriate grammar. However, only 18 (48.7%) contained relevant graphics and images, and 19 (51.35%) used an appropriate variety of material (i.e., text, graphics, tables, equations, audio and video), appropriate cover images (i.e., to attract attention and clearly portray the purpose of the materials), and appropriate illustrations and media (i.e., no auto-play and minimal use of animation). Usability. Most of the websites were judged to be easy to use. Thirty-six (97.3%) websites appeared to be well organized, had an index and a table of contents, made it easy to return to the previous page, and had a help feature and frequently asked questions. All the websites showed appropriate speed, taking less than 5 seconds for the pages to load, including text and images. Thirty-two (86.5%) websites had an internal search engine. In three of the websites, the search engine was enhanced by Google.
Accessibility. Under this domain, we evaluated the cultural match of the website, the appropriateness of color contrast, the findability of the content, and the availability of the website to people with disabilities. Most of the websites (94.6%) used appropriate language and images and presented examples in realistic and positive ways. We judged that 94.6% of the websites used appropriate color contrast, with font and background color contrasting, and were reader-friendly. We also determined that for most of the websites, the content was easy to find. On the other hand, we judged that only 32.4% of the websites complied with the World Wide Web Consortium 2018 guidelines, which means that the website is available to people with disabilities. Readability. The mean SMOG readability score for the 37 websites was 10 (standard deviation ±2.1) with a minimum of 6 and a maximum of 16. Only 2 websites had a readability score of 6, 1 website had a score of 7, and 8 websites had a score of 8. For websites that included more than one page or several PDF documents, we calculated the average score of all pages. Most of the websites (91.9%) used a conventional writing style, with active voice and simple sentences, and 94.6% consistently gave context before presenting new information.
Website scores. The maximum score that a website could have if all the answers to the items were "yes" was 72. All the websites except four had a final score lower than 50.
User experience. In S4 Table, we present the results of the user experience. All the websites met the expectations of the person who rated the user experience, and for more than 97% of the websites, the reviewer stated that she would return to the website and would recommend it to others. However, for 27% of the websites, the reviewer perceived that the website did not always use words that could be understood. Furthermore, for 13.5% of the websites, too little information about ICIs was given.

Discussion
To our knowledge, this is the first study to evaluate the content and quality of websites with educational material about ICI treatment. By assessing accuracy, completeness, technical elements, design and aesthetics, readability, usability, and accessibility, and navigation, we identified important differences between websites, allowing us to provide a comprehensive account of the current state of 37 patient-oriented websites on ICI treatment.
Most websites gave the complete definition, mechanism of action, and complications of ICI. More than 80% of the websites included a clear definition of the drugs and expanded on complications. However, only 20% of the websites gave information about the balance between benefits and harms. This is an important issue to address since previous studies have shown that presenting qualitative and quantitative information about benefits and risks leads to more accurate perceptions of drugs' adverse events and benefits [18]. In a randomized controlled trial that sought to evaluate whether qualitative information, quantitative information, or a combination of the two best communicates benefits and risks, the investigators observed that providing absolute frequencies and percentages may benefit consumers by improving their drug knowledge and changing their perceptions [19]. In the present study, more than 75% of the websites provided information about irAEs. However, less than half of the websites explained risk factors for developing irAEs, and only 13.5% provided information about the management of irAEs. Only three websites gave information about vaccine contraindications, adverse events, and efficacy in patients receiving ICIs. irAEs are adverse events of ICI treatment that occur through immune activation. Almost any tissue of the body can be affected, and some patients bear a greater risk of developing them. A meta-analysis published in 2017 showed that patients with melanoma being treated with ICIs had a higher frequency of gastrointestinal and skin irAEs [20]. Patients with preexisting autoimmune diseases could have a greater risk of irAEs and flares of their autoimmune disease. In a retrospective cohort study, irAEs or autoimmune disease flares occurred in 71% of cancer patients with preexisting autoimmune disorders who were treated with ICIs, but they were easily managed [21]. Similar findings were observed in other studies, with a slightly lower incidence of flares and adverse events [22,23]. Therefore, it is important for patients to receive clear and detailed information about risks and management of irAEs, which is largely lacking in current ICI websites.
Regarding the technical domain, most of the websites had general disclaimers, identification of the target audience, general disclosures, and contact information. More than 80% of the websites provided the date of last update. On the other hand, less than 10% of websites provided a clear hierarchy of evidence, and just 17% clearly stated the sources of the information. Almost all websites had good visual aspects, with appropriate typography and type of material. Furthermore, all the websites showed appropriate grammar. These findings indicate that the websites were strong overall in various technical aspects.
Concerningly, when we analyzed the readability of the websites, we observed that only two websites had a SMOG readability score of 6, and one website had a score of 7. The National Institutes of Health (NIH) recommend that health material be written at a grade 6-7 reading level. Our results echo those of previously published studies. In a study published in 2016, Hutchinson et al. observed that online educational material pertaining to 9 common internal medicine diagnoses used reading levels significantly above grade 6-7 [24]. Similarly, other studies have shown that the content of online educational material in other topics have on average a higher reading level than the NIH-recommended level [13,25]. Our findings indicate that more effort should be made to provide patients with less complex information on ICI treatment that is more understandable.
We also evaluated the user experience of a user. For all the websites, the reviewer was satisfied with the experience of using the website. We also noticed that more than 25% of the websites did not always use words that were easily understood. This result is similar to that of our previously published study that evaluated the quality of websites with educational material for patients with rheumatoid arthritis [16]. In that study, more than 35% of the websites were perceived to not always use easily understandable words.
Our study has some limitations. First, we used three search engines, but there are other search engines that might find other websites. However, Google is the most popular search engine in the United States, accounting for more than 85% of all searches done in the country [26]. Another point of concern is that we excluded websites from other countries. There could be well-developed websites with important information about ICI for patients from other countries that patients from the United States can have access and get information from. However, we decided not to include other countries' websites because the search engines we used account for IP address and would not prioritize the websites in the ranking of search results. Finally, the results of our study search would be difficult to replicate because of the complexity of the algorithms used by search engines to rank what comes up first in a search. Nevertheless, the results of our study give a picture of what is currently on the internet about ICIs for patients.
In conclusion, we have shown the strengths and weaknesses of the quality and content of current websites with information for cancer patients on ICIs, which will help organizations develop websites with detailed, high-quality information, presented in less-complex language. Patients could make better and more informed decisions if all the potential weaknesses found on the websites in our study are taken into consideration and addressed by the website's developers. Moreover, improving the readability of websites would make them easier to understand and use, especially for people with low health literacy skills. These individuals might feel better informed with a website that follows the standards outlined, which can lead to less uncertainty about their condition and improve their self-efficacy.
Supporting information S1